CI Fix bertopic testing - #8298

There's a more reliable way to do this AND to use a CUDA build of torch.

I recommend you follow how cugraph-gnn testing works, where we download (NOT install) a torch CUDA wheel:

https://github.com/rapidsai/cugraph-gnn/blob/28563c695245be55729dbc5a0a9227104b334f45/ci/test_wheel_cugraph-pyg.sh#L30

Then pass it as a requirement alongside the other things you're installing:

https://github.com/rapidsai/cugraph-gnn/blob/28563c695245be55729dbc5a0a9227104b334f45/ci/test_wheel_cugraph-pyg.sh#L19-L26

https://github.com/rapidsai/cugraph-gnn/blob/28563c695245be55729dbc5a0a9227104b334f45/ci/test_wheel_cugraph-pyg.sh#L60-L61

which would end with something roughly like this:

rapids-pip-retry install \
  --prefer-binary \
  "${LIBCUML_WHEELHOUSE}"/libcuml*.whl \
  "${CUML_WHEELHOUSE}"/cuml*.whl \
  /tmp/wherever-you-downloaded-torch/torch_*.whl \
  bertopic

Or if you really must still have separate install commands (if #7374 still applies here in 26.08), then the same thing but split like this:

rapids-pip-retry install \
  --no-deps \
  /tmp/wherever-you-downloaded-torch/torch_*.whl

rapids-pip-retry-install \
  --prefer-binary \
  "${LIBCUML_WHEELHOUSE}"/libcuml*.whl \
  "${CUML_WHEELHOUSE}"/cuml*.whl \
  bertopic

In general fewer separate pip install calls = reduced risk of a broken environment. And the more reliable way to force a particular file into an environment is to download it and pip install --no-deps it.

I'm fine for alternative approaches, especially if they have precedent somewhere else.

The reason I decided to install CPU only pytorch is that cuml and torch end up installing/depending on CUDA 12 and CUDA 13. There is --index-url https://download.pytorch.org/whl/cu126 so we could use that (via your "download first" approach)?

I could almost be convinced that running with what ever you get "the naive way" (separate pip install commands without the special torch index) is what we should test with. So what we have on main atm. The reason I am half convinced that we should leave it unchanged is that I think that configuration is a likely thing for a real world user to have installed (for better or worse). But it also feels a bit wrong to test that :-/

The "mixed CUDA versions" is a side comment in #8168

But it also feels a bit wrong to test that :-/

Yeah I'm sympathetic to this (wanting CI to match what users do), but I think in this particular case the job is testing things that are not cuML's responsibility, like the challenges of creating a pip environment with torch wheels and libraries that depend on them, and of incrementally creating a pip environment generally.

From my perspective, especially since this job we're talking about is just a smoke test, it'd be better to make the environment construction more reliable. You could think of that as demonstrating one example of how to get these things working together, which users facing issues in environments constructed other ways could be directed to.

@betatim @csadorf I pushed changes in 4a8f672

I think we can do something even simpler than the pip download approach I linked above, since this smoke test is the only place that cuml CI currently cares about a torch wheel dependency (as far as I can see).

I tested it locally and saw it work for both CUDA 12 and CUDA 13.

Added this job to PR CI temporarily so we can test here too, but it looks like our environments are still getting Cython 3.2.7 :/

#8287 (comment)

RAPIDS_TESTS_DIR isn't used anywhere in this script, it can be removed.